Systems and Methods for Planning Hair Transplantation

ABSTRACT

Systems and methods for creating a treatment plan for cosmetic procedures, for example, a hair transplantation procedure, are provided. The treatment plan allows user to provide input on a proposed hair element using a free-hand drawing in a two-dimensional plane, and for the proposed hair element to be generated and displayed on a three-dimensional model of a body surface. Various techniques and methods described in the application provide for improved planning of a natural looking hair.

RELATED APPLICATION DATA

The present application is continuation-in-part of U.S. application Ser.No. 13/844,317, filed Mar. 15, 2013, and entitled “Systems and Methodsfor Planning Hair Transplantation.

FIELD OF THE APPLICATION

This application relates generally to treatment planning systems,methods and method of their use. In particular, this application relatesto hair transplantation planning systems i.e., harvesting and/orimplantation of hair follicular units in a body surface, usually ascalp, and methods of their use.

BACKGROUND OF THE APPLICATION

Hair transplantation procedures are well-known, and typically involve(e.g., in a patient having male pattern baldness) harvesting donor hairgrafts from the side and back fringe areas (“donor areas”) of thepatient's scalp, and implanting the harvested follicular units in a baldarea (“recipient area”). Historically, the harvested grafts wererelatively large (3-5 mm), although more recently, the donor grafts maybe single follicular units, which are naturally occurring aggregates of1-3 (and much less commonly, 4-5) closely spaced hair follicles that aredistributed randomly over the surface of the scalp.

In one well-known hair transplantation process, a linear portion of thescalp is removed from a donor area using a scalpel cutting down into thefatty subcutaneous tissue. The strip is dissected (under a microscope)into component follicular units, which are then implanted into arecipient area in respective incisions or puncture holes made using aneedle. Forceps may be used to grasp and place the individual follicularunit grafts into the needle puncture locations, although otherinstruments and methods are known for performing this task.

U.S. Pat. No. 6,585,746 discloses a hair transplantation systemutilizing a robotic system, including a robotic arm and a hair follicleintroducer associated with the robotic arm. An imaging system is used toproduce a three-dimensional image of the patient's scalp, which is usedto plan the locations to receive hair grafts. The entire disclosure ofU.S. Pat. No. 6,585,746 is incorporated herein by reference.

No matter what type of hair transplant procedure is adopted, it is theaim of the physician to provide his patient with a natural looking headof hair. Currently physicians try and do this based on their experiencebased on procedures performed on prior patients, hoping that when theyharvest hair from donor areas of the patient's head and implant it intorecipient areas, they manage to do so in such a way that a naturallooking head of hair results. However, it is not likely that a realisticimage of what the patient might look like may be obtained based on priorexperience with other patients. Everyone's hair is different, it is notof the same thickness, texture or density, and hair does not lie or fallin the same way on heads of different people. Moreover, not everyone'shead shape or size is the same either, so there are many differenceswith respect to how the hair falls, and how it is parted too.Consequently, a result of a hair transplantation procedure performed onone patient does not necessarily look the same if the same procedure isperformed on a second patient. Currently, it is only possible for one tolook at the before and after photos of other people, and to guess whatthe outcome of a hair transplantation procedure might be.

Commonly assigned U.S. Pat. No. 7,806,121 and U.S. Pat. No. 8,104,480(hereinafter collectively, “Bodduluri”) illustrate systems and methodsfor planning transplantation of follicular units into a body surface ofthe patient. Both patents are incorporated by reference herein.

SUMMARY

A variety of systems and methods for planning various cosmetic anddermatological procedures, including hair transplantation, are providedin the present application. These procedures may be performed on thescalp, face and other skin and body surfaces.

According to one aspect of the present application, a method of planningfor transplantation of follicular units in a body surface is provided.The method comprising: generating and displaying on a three-dimensionalmodel of a body surface, at least two hair elements, each of therespective hair elements comprising one or more control points, thecontrol points having corresponding parameters associated therewith;associating, with a use of a processor, a first and a second of the atleast two hair elements with each other to form a hair group; generatingor modifying at least one site within or on the first hair element,based at least in part on the parameters of at least one of the one ormore control points of the second hair element. In some embodimentsproposed follicular units sites are automatically modified, or generatedand displayed based on the parameters of the control points of the hairgroup.

By the way of non-limiting examples, the hair element may be a hairpatch or a boundary curve (which may include a front hairline, or ahairline), the control points maybe located on a periphery or inside ofthe hair patch. The proposed sites may be hair harvesting sites,incision making sites, hair implantation sites, as well as various sitesfor placing and removing tattoos, or performing various cosmetic anddermatological procedures.

In some embodiments the parameter of the control point may be one ormore of a location, an orientation or a density. In the case of densitybeing the parameter, for example, in reference to hair transplantationand related procedures, the generation of the proposed sites may bebased on an inverse transform algorithm between the proposed site andone or more control points. The density parameter may be weighed basedon the distance of the associated control points from the proposed site.

In some embodiments, the method may further comprise projecting the3-dimensional coordinates of a hair element into a 2-dimensionalcoordinate system and subsequently interpolating a procedure site (e.g.,implantation site) in the 2-dimensional coordinate system. Theinterpolated implantation sites are then projected back into the3-dimensional coordinate system, in which they are depicted on the3-dimensional model of the patient. In some embodiments, theinterpolation method may take into consideration density, and utilize adensity parameter of a plurality of control points, wherein theplurality of control may comprise the one or more control points.

According to another aspect of the application, a method of planning fortransplantation of follicular units in a body surface is provided,comprising: generating and displaying on a three-dimensional model of abody surface, a hair patch comprising a set of control points on theperiphery thereof, the control points having corresponding parametersassociated therewith; creating additional control points within the hairpatch and/or on the periphery thereof, automatically generating anddisplaying a proposed follicular unit implantation site within or on theperiphery of the hair patch based on the parameters of the controlpoints and the additional control points.

According to another aspect of the current application, a system and amethod of generating proposed procedure sites based on densityinterpolation is provided. The method comprising: projecting a locationof a plurality of contour points of a body surface element in athree-dimensional coordinate system onto a two-dimensional plane;generating candidate procedure sites in the two-dimensional plane;determining an interpolated spacing value for each candidate proceduresite, based on a density parameter of a plurality of control pointsassociated with the body surface element; generating proposed proceduresites; projecting the two-dimensional location of the proposed proceduresites back to the three-dimensional coordinate system. In someembodiments, the method further comprises identifying candidateprocedure sites that are not available as procedure sites due to therequired spacing between the proposed procedure sites. In someembodiments, a formulation may be determined which associates densityand spacing of the procedure sites. Additionally, the step ofdetermining the interpolated spacing value may comprise utilizing thedetermined formulation.

According to another aspect of the application, a system for planning atransplantation procedure is provided, the system comprising: a userinterface including a user input device, at least one non-transitorystorage medium storing instructions, and a processor comprising one ormore modules for executing operations on image data, the one or moremodules comprising instructions for: generating and displaying on athree-dimensional model of a body surface, at least two hair elements,each of the respective hair elements comprising one or more controlpoints, the one or more control points having corresponding parametersassociated therewith; associating a first and a second of the at leasttwo hair elements with each other to form a hair group; generating ormodifying at least one site within or on the first hair element, basedat least in part on the parameters of at least one of the one or morecontrol points of the second hair element. In one embodiment the systemfurther comprises an image repository. The system and method of thepresent application is especially useful when implemented on, orintegrated with, an automated or computer-controlled system, forexample, an image-guided robotic system comprising a robotic arm. Thesystem may further include an imaging device to provide image datacontaining one or more images of the body surface and a tool forperforming a procedure. The system may comprise an interface adapted toreceive an image data containing images of a body surface, a processorprogrammed to perform various procedures and steps described inreference to various embodiments and methods of the present application,and a monitor configured to display a treatment plan as describedherein.

In another aspect of the present application, systems and methods ofplanning for the transplantation of follicular units in a body surfaceis provided. Such systems and methods comprise generating and displayinga three-dimensional model of a body surface, selecting from an imagerepository a template, and generating and displaying one or moreproposed hair elements based on the template. The step of generating maycomprise generating a portion of the template. The portion may comprisea boundary curve or a hair patch.

According to another aspect of the present application, a system andmethod of planning for modification of a body surface (including facialfeatures) is provided. Such systems and methods provide for associatingvarious body surface elements into the groups and modifying certainfeatures in a first body surface element based at least in part on oneor more parameters of the control points from the second body surfaceelement. In some embodiments modifications may be made automatically orwith the user input. By way of non-limiting examples, the body surfaceelements may be a mole, freckle, wrinkle, scars, facial features such aseyes, nose, eyes, eye-lids, lips, ears, chin, birthmarks, or facialdefects.

According to yet another aspect of the present application, a method ofplanning for transplantation of hair grafts is provided which utilizestwo-dimensional (2-D) user input of a proposed hair element from a freehand drawing. In some implementations, the method comprises receivingtwo-dimensional user input of a proposed hair element from a free-handdrawing; generating (for example, with a use of a processor) anddisplaying the proposed hair element on a 3-D model a body surface on adisplay device; and automatically generating a plurality of controlpoints based on the proposed hair element on the 3-D model. The methodmay further comprise automatically determining an orientation value ofat least one of the plurality of control points. The hair element maycomprise, for example, a hairline, a hair boundary curve, or a boundaryof a hair patch. Additionally, proposed follicular unit implantationsites or harvesting sites may be automatically generated and displayedbased on the plurality of control points and the orientations of theplurality of control points. Generation of the proposed hair element onthe 3-D model of the body surface on a display device may comprisedetermining if the received 2-D user input corresponds to an open or aclosed loop; and generating and displaying a hair line if the determinedinput corresponds to an open loop, and generating and displaying a hairpatch if the determined input corresponds to a closed loop. In someembodiments, the method further comprises receiving additional 2-D userinput corresponding, for example, to orientation information, in theform of a free-hand drawing, wherein the additional 2-D user input isused to automatically generate proposed follicular unit implantationsites. In some embodiments the additional 2D user input comprises one ormore curves drawn within an area outlined by a boundary of a hair patch.In yet other embodiments, the method may further comprise identifying alocation of a center of a hair whorl with respect to the hair element.In one aspect of the application, the method further comprisesdisplaying a 3D model of a body surface on a display device.

According to another aspect of the application, a system for planning atransplantation procedure is provided, the system comprising: a userinterface including a user input device, at least one non-transitorystorage medium storing instructions, and a processor comprising one ormore modules for executing operations on image data, the one or moremodules comprising instructions for: receiving two-dimensional (2-D)user input of a proposed hair element from a free-hand drawing;generating and displaying the proposed hair element on a 3-D model abody surface on a display device; and automatically generating aplurality of control points based on the proposed hair element on the3-D model. In some embodiments, the system further comprises one or moremodules comprising instructions for determining if the received 2-D userinput corresponds to an open or a closed loop; and generating anddisplaying a hair line if the determined input corresponds to an openloop, and generating and displaying a hair patch if the determined inputcorresponds to an closed loop.

According to a further aspect of the application, a method ofautomatically creating a plan for a hair whorl on a body surface isprovided. The method comprises: identifying a hair boundary curve, thehair boundary curve having a plurality of control points; identifying ahair whorl center; automatically assigning an initial orientation valueto each of the plurality of control points, the initial orientationvalue based on the orientation of a virtual line from the whorl centerto each of the respective plurality of control points; and automaticallygenerating proposed follicular unit implantation sites within an areaoutlined by the hair boundary curve based on the plurality of controlpoints and their corresponding assigned initial orientation values. Themethod may further comprise modifying an orientation of an automaticallygenerated proposed follicular unit implantation site based on itsdistance from the hair whorl center. In some embodiments themodification is such that it has a greater deviation from the initialorientation value the further it is from the hair whorl center, and alesser deviation from the initial orientation value the closer it is tothe hair whorl center.

According to another aspect of the application, a system forautomatically creating a plan for a hair whorl on a body surface isprovided, the system comprising: a user interface including a user inputdevice, at least one non-transitory storage medium storing instructions,and a processor comprising one or more modules for executing operationson image data, the one or more modules comprising instructions for:identifying a hair boundary curve, the hair boundary curve having aplurality of control points; identifying a hair whorl center;automatically assigning an initial orientation value to each of theplurality of control points, the initial orientation value based on theorientation of a virtual line from the hair whorl center to each of therespective plurality of control points; and automatically generatingproposed follicular unit implantation sites within the area outlined bythe hair boundary curve based on the plurality of control points andtheir corresponding assigned initial orientation values.

According to yet a further aspect of the application, a method ofplanning for transplantation of follicular units or hair grafts in abody surface is provided, the method comprising: receiving one or moretwo-dimensional (2-D) curves from a free-hand drawing; automaticallyassigning at least two control points to each of the one or more curves;automatically determining an initial orientation value for each of thecontrol points based on an angle of a tangent to the one or more curvesat each respective control point; and automatically generating proposedfollicular unit implantation or harvesting sites based on the at leasttwo control points of the one or more curves and their correspondingautomatically assigned initial orientations. A system and a processorcorresponding to the above-mentioned method is also provided.

Other and further objects and advantages of the invention will becomeapparent from the following detailed description when read in view ofthe accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

Features and advantages of the embodiments described herein will becomeappreciated as the same become better understood with reference to thespecification, claims, and appended drawings wherein:

FIG. 1 is a flow chart illustrating an example of a general method forgenerating a treatment plan for use with a hair transplantationprocedure.

FIGS. 2 a and 2 b are images of a front and back view of a person'shead.

FIG. 3 depicts an example of a proposed hair implantation elementaccording to one embodiment of the application.

FIGS. 4 and 5 are examples of the representative monitor views generatedas part of a treatment planning illustrating the use of two differenttemplates.

FIG. 6 a depicts an example of how control points over two hair elementsmay influence a proposed site.

FIG. 6 b depicts an example of how control points over three hairelements may influence a proposed site.

FIG. 7 a depicts an example of how hair whorls may be generated.

FIG. 7 b depicts another example of how hair whorls may be generated.

FIG. 7 c is a flow chart illustrating an example of a method forgenerating a hair whorl.

FIG. 8 illustrates by example a person's head on which three overlappinghair elements are superimposed.

FIGS. 9 a and 9 b depict the three overlapping proposed hair elements ofFIG. 8.

FIG. 10 illustrates an example of one hair element completelyoverlapping another hair element.

FIG. 11 illustrates an example of a proposed hairline adjacent to aproposed hair patch.

FIG. 12 is a flow chart illustrating an example of how interpolatedsites can be generated for a hairline.

FIG. 13 depicts an example of an interpolated hairline.

FIG. 14 is a flow chart illustrating an example of how sites can begenerated within a hair boundary.

FIG. 15 illustrates one example of an embodiment of a 2-D projection ofa hair patch.

FIG. 16 illustrates another embodiment of a 2-D projection of a hairpatch including proposed implantation sites.

FIGS. 17 a and 17 b illustrate an example of a free-hand drawn hairline, its associated control points, and proposed follicular unitimplantation sites.

FIGS. 18 a and 18 b illustrate an example of a free-hand drawn hairpatch, its associated control points, and proposed follicular unitimplantation sites.

FIGS. 19 a and 19 b illustrate an example of how free-hand drawn curvescan influence proposed follicular unit implantation sites.

FIG. 20 is a flow chart illustrating an example of a general method forgenerating treatment plan utilizing free-hand drawing.

FIGS. 21 a and 21 b illustrate a representation of the patient's head,including hair regions with proposed implantation sites, and tensioningdevice representations.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following Detailed Description reference is made to theaccompanying drawings that show by way of illustration specificembodiments in which the application may be practiced. In this regard,directional terms, such as “inner”, “front”, “away”, “top”, “right”,“left”, etc., are used with reference to the orientation of theFigure(s) being described. Because components or embodiments of thepresent application can be positioned in a number of differentorientations, and the methods can be carried out in a number ofdifferent ways, the directional terminology is used for purposes ofillustration and is in no way limiting. Also, the terms “coupled,” or“attached,” or “connected,” or “mounted” as used herein, means directlyor indirectly coupled, attached, connected, integrated, or mounted, forexample, through one or more intervening components. It is to beunderstood that other embodiments may be utilized and structural orlogical changes may be made without departing from the scope of thepresent application. The following Detailed Description, therefore, isnot to be taken in a limiting sense, and the scope of the presentapplication is defined by the appended claims.

When planning a hair transplantation procedure for a particular patientthe physician has to take many parameters that vary from one patient toanother into consideration. For example, hair density and the variationof density throughout different regions of the patient's scalp, thecaliber of hair which may also vary throughout the patient's scalp, themanner in which the patient styles and/or combs his hair. Having takenthese parameters into consideration, it is however not possible to knowwhether the hair that is implanted will match the already existing hair,give a natural-looking appearance, or look as if it has beentransplanted. It is also not possible to know how the area from whichhair is being harvested will look like after hair has been removed orharvested.

Natural looking randomness is important not only for the criticalhairline element but in the balance of the recipient sites, and thedonor site too. To this end, each hair element, comprising, for example,hairlines, hair patches and the donor area should not be designed orplanned independent from each other. In planning a hair transplant, thephysician has to ensure that his plan offers the best outcome based uponthe existing hair that exists on the patient's head, for example, thedensity or the distribution of existing hair, its spacing, angles, howthe patient generally parts and/or combs his hair, or whatever elsecontributes to making the hair transplant look natural. One aspect ofthe current application enables such a natural looking randomness to bemaintained, for example, throughout the patient's scalp. In addition,another feature of the current application enables a physician to obtainan image of what his/her patient may look like at various stages during,as well as, after a hair transplantation procedure is completed, basedupon an image of their head, and their actual hair. Utilization ofvarious aspects of the current application provides not only an image ofwhat the newly implanted hair may look like, but of what the areas fromwhich the donated hair will look like once the hair has been harvestedfrom the donor element(s). According to another aspect of the currentapplication a means is also provided by which a physician is able topropose potential visual appearances and aesthetic outcomes to apatient, explaining the various functional and/or aesthetic advantagesand disadvantages of the potential treatment plans, along with adiscussion of the time and/or cost associated with each proposedtreatment plan. In this manner the patient is able to see what he/shemay look like in each of the scenarios discussed, which reduces thechances of a patient misunderstanding what the physician may be tryingto convey to a patient. In yet a further aspect of the currentapplication a means is also provided by which the physician is able toplan one or more sessions of transplant surgery, optimizing the processto potentially reduce the number of sessions required to complete a hairtransplant. This application provides the physician with a tool toenable him or her to ensure that there is, for example, sufficient hairin the donor areas, and hair of the correct type and caliber to providethe look that the patient is hoping for. If there is not, the currentapplication provide the physician with a tool to enable him/her toillustrate to the patient what possible hair transplantation options areavailable to him or her based on the number, type, and caliber of hairthey have available in the donor areas. The system and methods describedin the present application also provide the physician with a means forplanning the hair transplantation process, the number of sessions etc.

It should be noted that although the application is particularly usefulin hair harvesting and implantation, it is not limited to hairtransplantation. The application may also be beneficial to otherprocedures that require a model of the patient's body surface and parts,including, facial and head features, for example, various cosmetic anddermatological procedures involving treatment planning (e.g., plasticsurgery, wrinkle removal or reduction, injections of cosmeticsubstances, skin grafting procedures, correction or removal of birthmark defects, facial reconstruction, rhinoplasty, contouring of the eyesor lips, remodeling of ears, nose, eye-lids or chins, facialrejuvenation, laser skin resurfacing, skin tightening, etc.) may benefitfrom the system and method of the applications described herein. Oneexample of applicability of the application is in diagnostic skinimaging for cosmetic or other medical purposes, for example skingrafting or tattoo removal. For convenience of description, thefollowing description will be discussed by example in reference to hairtransplantation procedures. It should be noted, however, that suchdescription is for the purposes of illustration and example only and isnot intended to be exhaustive or limiting.

In accordance with one embodiment of the application, a treatment planfor a hair transplantation procedure may be created at least partiallyby a computer. The treatment plan automatically modifies, or generatesone or more proposed follicular unit sites within a first hair elementbased on one or more parameters of hair follicles or follicular units inone or more second hair elements. In other embodiment of theapplication, the treatment plan automatically modifies, or generatesproposed follicular unit sites within a first hair element based on oneor more parameters of follicular units in the hair group, the hair groupcomprising a first and one or more second hair elements. In either case,the parameter information between two hair regions is shared, thusproviding a tool to facilitate a more natural looking hairtransplantation result. The treatment plan may take into considerationone or more parameters, including, but not limited to: a location of oneor more hair elements, the geometric profile of hair elements, thenumber and/or type of follicular units to be harvested/implanted, adegree of randomness associated with particular harvest/implantlocations, spacing between adjacent implant locations, depth offollicle, depth of implant, orientation of follicles or follicularunits, patient identification (for example hair color, ethnic origin,age etc.), marker location(s), and/or the density of harvest/implantsites. In some embodiments, information about a particular follicularunit or combination of follicular units can be obtained from a databaseof follicular units. This database may contain information specific tothe patient in question or to patients in general. The database maycontain information categorized by ethnic origin, gender, and/or age,for example. In yet a further embodiment, the treatment plan may begenerated or modified based on control points on or within the firstand/or second hair elements. The control points can be individualpoints, lines, shapes, or other markers that can be used to provide abasis for the generation or modification of follicular unit sites withinthe first hair element.

After the treatment plan has been created at least partially by thecomputer, the user can accept or alter the treatment plan. Once thetreatment plan meets the expectation of the physician, user and/or thepatient, the user may register the treatment plan with a patient. Insome embodiments, this may be accomplished by using one or more camerasto identify one or more markers that could be positioned directly on thepatient or on a device used on the patient. The marker may be areflector that is secured to the patient, an ink mark drawn on thepatient, or an anatomy of the patient. Alternatively the marker may be,for example, a marking on a skin tensioning device that may be utilizedby the physician in the hair transplantation procedure. The identifiedmarker(s) may be used to determine a position and/or orientation of anelement on the patient.

Hair transplantation procedures that are carried out using automated(including robotic) systems or computer-controlled systems have beendescribed, for example, in U.S. Pat. No. 7,962,192, commonly owned bythe assignee of the present application, which is incorporated herein byreference. After the robotic system has been initiated and calibrated,image data of the body surface is acquired and processed by the systemcomputer to identify objects, in particular follicular units in a donorelement, for example, on a human scalp. From images of this element ofinterest, image segmentation and screening software residing in thecomputer identifies and selects particular follicular units of interestfor harvesting from the scalp.

In accordance with various embodiments of the application, a system forplanning a procedure for transplantation of follicular units in a bodysurface (e.g., a scalp) of a patient may comprise a user interface,processor (e.g., software-controlled), a monitor, and at least one inputdevice. These components are common to virtually all modern computersystems, whether a stand alone (e.g., “personal”) computer system, or ina system employing a centralized server with multiple remoteterminal(s). It will be appreciated that embodiments of the planningsystem are preferably (if not exclusively from a practical point ofview) software implemented, and may be run on any computer system havingthe basic components (processor, monitor, input device), so long as suchcomputer system is equipped with sufficient available memory and anappropriate graphic generation and display capability. The computingsystem may include one or more processing units, one or morenon-transitory storage media (which may take the form of, but is notlimited to, a magnetic storage medium; optical storage medium;magneto-optical storage medium; read only memory; random access memory;erasable programmable memory; flash memory; and so on), and/or one ormore input and/or output components for transmitting output to and/orreceiving input from one or more other components (such as one or moredisplays, touch screens, keyboards, mice, track pads, track balls,styluses, pens, printers, speakers, cameras, video cameras, and so on).The processing unit may comprise one or more modules to executeinstructions stored in the storage medium in order to perform one ormore computing device functions, such as one or more treatment planningmethods. The system or the processing unit may additionally include animage repository, the image repository comprising templates, images ofone or more patients and/or images of portions of templates or patients.The system can be configured to implement all the methodologies,processes and techniques described herein.

Although it may be suggested that the computing system includeparticular components arranged in a particular configuration, it isunderstood that this is for the purposes of example. In variousimplementations, the computing system may include any number ofcomputing system components (such as one or more busses, displays,networking components, dedicated image processors, co-processors,memories, hard drives, ports, graphics adapters, and so on) arranged indifferent configurations without departing from the scope of the presentdisclosure. For example, in one or more implementations the computingsystem may include multiple cameras and/or video cameras arranged tocapture images and/or video of the same scene. By way of anotherexample, in various implementations the computing system may include oneor more interfaces for controlling machinery such as automated and/orcomputer-assisted surgical machinery.

It will also be appreciated that embodiments of the application may beimplemented over the internet, e.g., with a user of such systememploying his or her home computer as at least a part of the userinterface (monitor and input device) that interacts with a remote serveror computer. In such an internet-based planning system, the softwarethat implements and controls the user interface may reside in whole orpart on the user's computer or OD the remote server/computer, preferablytransparent to the user. In one such embodiment, the remote serverdownloads one or more software modules to the user's computer fortemporary or permanent use.

Exemplary embodiments of a software implemented and controlled userinterface for planning a follicular unit transplantation procedure willnow be described in conjunction with the accompanying figures. It willbe appreciated that various and multiple variations of the describedembodiments may be implemented without departing from the general scopeof the application, which is set forth in the appended claims.

FIG. 1 is a block diagram illustrating an example of a generalmethodology of a treatment planning procedure according to the presentdisclosure. As a preliminary matter, at step 110, image data isacquired, for example, by using an image acquisition or imaging device,or from pre-existing data stored in the computer system's memory, or anyother technique known in the art. For example, in some embodiments, theimage data can be retrieved from the patient's electronic records orgeneral patient data. In other embodiments, the image data can becreated real-time, using a digital camera. The image data may compriseone or more images, including sufficient 2-D images to enable an imageprocessing unit of the computer system to create a 3-D image.

In step 120, a processor or an image processor, processes and recordsinformation associated with the image data. Such information comprising,for example, the location and orientation of existing follicular unitsor hair follicles, information pertaining to scars, moles, freckles,wrinkles, facial features, tattoos or any other such body surfacefeatures.

In step 130, based on the information determined at step 120, theprocessor proposes hair elements, or body surface elements, includingfacial elements (and consisting of the proposed modification of existingbody surface features) and one or more presentation components in thetreatment planning system, such as a monitor, are used to visuallypresent information indicative of the proposed hair elements/bodysurface elements. In reference to hair transplantation, the proposedhair elements may comprise, for example, a hairline or a hair patch. Theproposed hair elements may comprise elements from which follicular unitsare to be harvested from or implanted into. In one embodiment, theproposed hair elements comprise a proposed boundary curve and/orproposed harvesting/implantation sites. In some embodiments, theproposed hair elements are proposed based on the image data alone. Inother embodiments the hair elements are proposed based additionally ontemplates residing in the image repositories of the computer system.These templates may be automatically selected by the computer systembased on the image data processed, or alternatively they may be selectedby the user, or by a combination of both. In some embodiments once atemplate has been selected, portions of the template may further beselected. In yet another embodiment, the proposed hair elements may beidentified or generated by the user, for example, by freehand drawing,using for example a mouse, stylus, pen or line tool on a touch-enableddevice, tablet, or other such similar device.

Optionally in step 140, executable instructions may be provided toenable the user to modify the proposed hair elements/body surfaceelements by modifying the proposed boundary curve and/or the proposedharvesting/implanting sites based upon their analysis of the displayedimage. The proposed boundary curve and/or the proposedharvesting/implanting sites may comprise control points to assist in themodification process. In one embodiment, the user modifies the proposedhair boundary curve or proposed harvesting or implanting sites in a 2-Dview, and the computer system converts the 2-D modification into a 3-Dmodification based on the 3D data, such as data from a 3D mesh model.

In step 150, two or more hair elements/body surface elements areassociated to form a group, such as hair group. Sites are modified orgenerated in step 160 based on the defined hair groups. Take, forexample, a hair group comprising two hair elements. In one example, oneor more proposed follicular unit sites in a first hair element may benow modified or generated based on one or more parameters of follicularunits in the second hair element that is in the same hair group with thefirst element. In another example, one or more proposed follicular unitsites in a first hair element are modified or generated based on one ormore parameters of follicular units in the hair group as a whole.

In the event that the user wishes to associate a different combinationof hair elements to modify or generate more proposed sites, steps 150and 160 may be repeated until all desired combinations have beenassociated and the desired proposed sites modified or generated.

In step 170, the treatment plan is accepted and, optionally, in step 180a grid plan (described in more detail below) may be generated.

Although the method 100 is illustrated and described as includingspecific operations performed in a specific order, it is understood thatthis is for purposes of example. In various implementations, someoperations may be performed in another order without departing from thescope of the present disclosure. In other implementations, only a subsetof the various operations may be required, again without departing fromthe scope of the present application.

At any stage of the procedure, the proposed plan may be saved forfurther modification, editing or updating at a later date. This may bebeneficial if after execution of a particular session of the treatmentplan, the patient's hair does not grow as predicted, requiringmodification of the treatment plan. Or, after execution of a particularsession of the treatment plan, the patient has a change of mind, orcannot afford to have the entire procedure performed as planned,requiring modification to address his financial concerns, while at thesame time providing him with a hair transplant that has a naturallooking appearance.

FIGS. 2 a and 2 b, illustrate an example of an embodiment of how imagedata is acquired (step 110). Images 205 a and 205 b are acquired of abody surface of a patient, for example, various views of the patient'shead (and, in particular, scalp), for which the subject transplantationprocedure is being planned. By way of non-limiting example, the imagesmay be acquired using a hand held digital camera, or even a mobiletelephone camera, and input through the user interface of the planningsystem, in accordance with well-known and available technology fortransmitting digital image data. It is not necessary to include imagesof every portion of the patient's head, since it is known to thoseskilled in the art that modeling software can generate a sufficientlyaccurate three-dimensional surface model of the head/scalp from just afew views, for example, from four (4) to six (6) views from differentdirections and sides, including, for example, from the top. Dependent onthe purpose of the treatment plan (e.g., facial cosmetic procedure, orplanning hair transplantation on the scalp), the minimum number andangles/directions of the desired views may be adjusted to achievesufficiently accurate model. The acquired images are then processed togenerate a three-dimensional model of the patient's head (e.g., scalp)using modeling software. It will be appreciated that any suitablesoftware program may be used.

In an alternate embodiment in which acquired images of the patient'shead/scalp (or other applicable body surface) are not provided, themodeling software is configured to generate (by selecting menu option“generate”) a three-dimensional model based on inputs relating toselected characteristics, such as ethnic origin or race, and/or othercharacteristics selected through one or more menus, some objective(e.g., gender, age) and others purely subjective (e.g., attractive,symmetrical features, long forehead).

In some embodiments, further input information is provided to theplanning system in addition to generating the body surface model. Forexample, particular features specific to hair follicles (e.g., color orcoarseness) may be derived from the images and/or input through the userinterface. Further information may be either user input or determinedfrom the acquired images using image processing, such as geometric shapeof the body surface (e.g., patient's head), existing hairlines, theexistence of scars or moles, the location of where follicles havepreviously been harvested from or implanted into, and a number of eachtype (i.e., single or multiple follicle) and color (e.g., dark, light,gray, etc.) of follicular unit that are available for harvesting. Itwill be appreciated that the three dimensional model can alternativelybe generated by a number of other methods, for example, using a 3D laserscanner and/or by stitching multiple digital images together. The systemof embodiments of the present application will use the three dimensionalinformation in the same way, without regard to how it is generated.

Whether from the acquired images, or through other descriptive featureinputs, the modeling software generates and displays on the userinterface monitor of the planning system a three-dimensional model ofthe patient's head/scalp. With reference to hair transplantation, thismodel will represent the patient's head/scalp, and therefore willtypically exhibit characteristic of male/female baldness patterns,comprising of bald regions 210, follicular unit populated regions 215and, and/or regions which are more or less densely populated thanothers. The populated regions 215 will comprise follicular units whichare grouped to form regions of hair which border less populated regions,or bald regions. These regions of hair may have geometry associated withthem. The follicular units have various parameters such as type,caliber, orientation with respect to the scalp, spacing between adjacentfollicular units, height of follicles above the scalp, for example.

Based on these characteristics, information, and/or one or more physicalfeatures determined from processing the acquired images, the modelingsoftware automatically generates and the planning system displays on theuser interface monitor of the planning system, a visual representationof one or more proposed elements 220 for follicular units sites as shownin FIG. 3 (step 130 of the flow chart of FIG. 1). According to oneaspect of the present disclosure, the planning system may automaticallypropose a treatment plan. For example, hair elements may be proposedinto which hair follicles may be implanted to create a natural look forthe patient. This may be accomplished by mapping anatomical elementsdetermined from the acquired imaged superimposing the proposed hairelements, and displaying them on the user interface monitor. Suchanatomical elements comprising, for example, the geometric shape of thehair populated regions and the geometric shape of the bald regions,follicular unit populated regions, the location of the eyes, ears andnose of the patient. One or more proposed elements for follicular unitsites may, for example, as illustrated in FIG. 3 be proposed in a baldregion 210 and comprise a boundary curve 230 and optionally one or moreproposed implantation sites 240. In this particular illustration controlpoints 250 are illustrated. These control points 250 may be generated bycomputer system and/or determined by the user. Typically the controlpoints lie on the boundary curve, though they may reside within the hairelement 220 or adjacent thereto. The user may optionally modify theboundary curve 230 by using a conventional click and drag motion of acomputer mouse (step 140) to drag the boundary curve 230, or drag thecontrol points 250. The system is configured such that the operation ofdragging the control points 250 is interpolated to every point along theboundary curve 230 between adjacent control points. According to anotheraspect of the disclosure, the user may manually propose a treatment plan(including a plan for a donor area where hair grafts will be harvestedfrom and a plan for recipient area where hair grafts will be implantedinto), using a stylus and touch screen display, for example, proposing afreehand drawn two-dimensional (2-D) plan which the computer system andits various processing units convert to a three-dimensional (3-D) plan.This aspect will be described in greater detail later in the disclosure.

According to some embodiments, a treatment plan may be generated basedat least in part on imaging data such as images from one or more imagerepositories. For example, the planning system may map anatomicalelements determined from the acquired images to model heads or templatesthat are within the image repositories to automatically propose atreatment plan. In some embodiments, the user may select (for example,by moving a cursor and clicking a mouse, using the arrows on a key padto move a selection box, or by touching with a stylus) an image ortemplate from patient information repositories containing images fromone or more patients. For example, the template may identify the finaloutcome or the subsequent stage that the patient wishes to reach interms of his/her hair transplantation treatment process. These templatesmay, for example, suggest different hair style outcomes that may beachievable. These templates may comprise templates for men, women, boys,or girls; templates corresponding to various ethnic origins; templatescorresponding to various head shapes; and/or templates corresponding tovarious degrees of hair loss, such as a template based on the Norwoodscale of hair loss. For example, a template exhibiting a male Caucasianpatient with hair receding mildly in a wedge-shaped pattern may be usedfor a patient with similar wedge-shaped pattern of baldness. Havingselected the most appropriate template or combination of templates, themodeling software generates and the planning system maps the selectedtemplate(s) to the acquired image data, and displays on the userinterface monitor of the planning system, a three-dimensional model of apatient's head/scalp, adding in or superimposing one or more elements asdictated by the selected template using, for example, a best-fitalgorithm. That is, a visual representation of the patients existinghair regions, including hair elements into which it is desired thatfollicular units be implanted. These one or more elements are generatedsuch that they are scaled to fit on the three dimensional model of thepatient's head/scalp. Optionally, the one or more elements may bemodified in shape and size to fit the three dimensional model of thepatient's head/scalp. Such modification may require, for example, themodeling software to place a sizing box around each, a subset of, or allof the one or more elements. The sizing box allows the user to vary thesize and shape of the one or more elements in a manner well known tothose in various arts. For example, selecting and dragging any side ofthe box by stretching/compressing the respective side, and/or byselecting a corner and stretching/compressing the whole box.Modification may also comprise rotating each, a subset of, or all of theone or more elements.

In another implementation, the images from the repositories can be usedto identify regions of interest through segmentation, contouring, or thelike. For example, images from the image repositories may identify hairelements which are proposed to be added to the three-dimensional modelof a patient's head/scalp. For example, the user may select apredetermined hair harvesting/implantation element from a range ofpossible options. This is illustrated in FIG. 4 which shows a view on auser interface monitor of the planning system, on which a threedimensional model 305 of the patient's head/scalp is displayed on theright hand side (as viewed by the reader). On the left, a template 310is illustrated, the template comprising several selectable elements intowhich it may be proposed that follicular units be implanted. In thisparticular example, the user may select one or more of a frontal element320, a forelock element 325, a mid-scalp element 330, or a crown element335 as element(s) into which hair is to be implanted. The donor area 340(shown on the left template side) is not a selectable element, thisbeing the element from which follicular units will be harvested tofulfill the implantation requirement. Having selected the desiredelements into which follicular units are to be implanted, the modelingsoftware generates and the planning system displays on the userinterface monitor of the planning system representations of the selectedelements on the three dimensional model 305 of the patient's head/scalp.In the illustrated example, on the image appearing on the right-handside of the user interface monitor, outlines of proposed hair elementscomprising control points on the periphery thereof can be seen,indicating that the frontal element 320, the forelock element 325 andthe crown element 335 have been selected by the user. In this particularexample, representations are generated such that they are scaled to fiton the three dimensional model of the patient's head/scalp. As shown onthe right hand side of FIG. 4, the representations may comprise controlpoints 345 which can be selected and dragged, thus enabling the user tomodify the shape and sized of the representations on the threedimensional model of the patient's head/scalp.

FIG. 5 illustrates another example in which once again a threedimensional model 405 of the patient's head/scalp is displayed on theright hand side (as viewed by the reader), and a template 410 on theleft illustrating one fewer elements than that illustrated in FIG. 4. Inthis example the user may only select from a template 410 one or more ofa frontal element 420, a mid-scalp element, 430 or a crown element 435as element(s) into which hair is to be implanted. This limitation ofelements available for selection may be due to a prior choice of theuser (for example, the user having selected this template from aselection of templates), the session of the hair transplantationprocedure (for example the first session of three transplantationsessions), the limited number of donor follicular units, financialreasons, the physician's choice, or based on prior transplantation dataetc. In this instance, if the user has selected that hair to betransplanted only into the frontal element, then the modeling softwaregenerates and the planning system displays on the user interface monitorof the planning system a representation of the frontal implant element420 on the three dimensional model 405 of the patient's head/scalp. Asillustrated, the representation comprises control points 445 which canbe selected and dragged, thus enabling the user to modify the shape andsized of the representations on the three dimensional model 405 of thepatient's head/scalp.

According to yet another aspect of the application, having selected froma range of possible templates a template which best identifies his/herpatient, the modeling software generates and planning system displays onthe user interface monitor of the planning system, a selection oftreatment plan options. These treatment plan options may, for example,include a high density follicular unit implantation in the frontalelement only, a lower density follicular unit implantation in thefrontal element only, follicular unit implantation in the frontal andmid-scalp elements, or perhaps follicular unit implantation throughoutthe scalp of the patient, for example. It will be apparent to the readerthat the number of options available is numerous and as such allexamples are not described herein. In another embodiment, the modelingsoftware may additionally provide an indication of which of the manyoptions are actually available to the patient, for example by making itimpossible to select certain options. For example, if an option of ahigh density follicular unit implantation throughout the patient's scalpis not possible due to the lack of donor follicular units that can beharvested, for example, the modeling software will not enable thatspecific treatment option, and will not make it an option that can beselected by the user. To implement this option, it will be apparent thatthe computer system will have had to attain information pertaining tothe number of, and optionally the type, of follicular units availablefor implantation. This information may be input by the user into thetreatment planning system, or be provided by the treatment planningsystem itself, in analyzing the image data, and determining how manyfollicular units are available for harvesting without detriment to theaesthetic appearance of the hair in a donor area on the patient's head.In an alternative configuration, the treatment planning system may beconfigured to plan the harvesting of follicular units from the existingfollicular units, and based on the number and/or type of follicularunits proposed to be harvested, can utilize this number and/or type ingenerating a proposed implantation plan. It will be apparent to thereader that the options that are enabled to be selected by the user willvary based on context.

However generated, the modeling software generates and the planningsystem may display on the user interface at least one proposed hairelement, for example a particular hair implantation site. Havingselected the hair elements within which, for example, follicular unitimplantation is desired, the treatment plan may be further customized.

According to another aspect, the present application also provides atreatment plan which automatically modifies, or generates proposedfollicular unit sites within a first hair element based on one or moreparameters of follicular units in one or more second hair elements. Thisaspect of the application will now be described with reference to FIGS.4 and 6 a. It was explained above (in relation to FIG. 4) that the userhad selected hair implantation elements 320, 325 and 335. Let us assumethat the three dimensional model of the patient's head/scalp 305 asdisplayed on the right hand side (as viewed by the reader) representsthe current state of the patient's head. A state in which the frontaland forelock element are more densely populated with hair than the crownelement. The user may select, for example, a first element, the crownelement 335, as an element which the user would like hair grafts to beimplanted into, the implantation carried out such that the resultingdensity of follicular unit implants provides for a density substantiallyequivalent to that of the forelock element 325. The user thereforedesires that a treatment plan be provided such that the follicular unitimplantation sites within the first element, the crown element 335, aregenerated based on a parameter (for example the density) of thefollicular units in a second element, the forelock element 325.

This scenario is depicted in greater detail in FIG. 6 a. It should beapparent to the reader that although the shape and details of thevarious elements have been simplified, the regions depicted as 325 and335 in FIG. 4 generally correspond to the regions depicted as 325 and335 in FIG. 6 a. The first of the elements 335 comprises a set of one ormore control points that are associated with it, illustrated by the fivecontrol points, P₀, P₁, P₂, P₃ and P₄. The second element 325 alsocomprises a set of one or more control points that are associated withit, illustrated by the four control points Q₀, Q₁, Q₂, and Q₃. Thesecontrol points may be determined or proposed by the treatment planningsystem. Alternatively they may be indicated as control points by theuser by simply moving a cursor on the screen to the location of theproposed control point, and clicking a button on a mouse to create acontrol point. In another alternative, control points proposed by thetreatment planning system may be modified by the user, once again usingthe drag feature and a mouse for example, or additional control pointsmay be added to those proposed by the system. The control points may bebased on elements within the image of the patient's body surface, suchas follicular units or hair follicles, or be based on input provided bythe user, virtual. The properties or parameters associated with anycontrol point may be determined from the image data, provided by thetreatment planning system, or provided by the user. These parameters, inreference to hair, may include, for example, information on thefollicular unit type, the length of hair follicles, the angle of hairfollicles or a density value associated with a hair follicle orfollicular unit (typically based on determining the number of hairswithin a specified distance of the control point). With reference tocosmetic or dermatologic treatments, these parameters may comprise skincoloration, skin texture, skin tautness, facial feature dimensions,facial topology etc. Once created, in this aspect of the application,rather than the parameter(s) of the proposed implantation site 510 beingbased on only the five control points P₀, P₁, P₂, P₃ and P₄ whichsurround it, as depicted by the solid arrows, the parameter(s) of theproposed implantation site 510 are based on all nine control points Q₀,Q₁, Q₂, Q₃, P₀, P₁, P₂, P₃ and P₄, as depicted by the solid and dashedarrows. The parameter(s) of the proposed implantation site 510 may beinterpolated from the parameter(s) of the nine control points.

It will be appreciated that there are numerous methods of interpolationthat may be utilized for this application. One example of aninterpolation technique is one in which the proposed hair site parameteris based on an inverse distance transform algorithm. This method ofinterpolation calculates the distance between the point to beinterpolated (e.g., 510) and each of the respective control points, Q₀,Q₁, Q₂, Q₃, P₀, P₁, P₂, P₃ and P₄, and applies a weighting to each ofthem based on the distance from the point to be interpolated (510). Inthis manner, the control P₁ will have a greater influence on theinterpolated value than the control points P₀ and P₂. Similarly controlpoints Q₂ and Q₃ will have the least influence on the value of theinterpolated point 510. For this algorithm, each of the control pointsmay be utilized and provide a contribution to the determined value ofthe point 510 to be interpolated. In an alternative, only those within acertain distance from the point to be interpolated may be considered or,for example, only the two closest points. The interpolated value appliedto the point to be interpolated 510 may be based on a linearinterpolation of the control points, a polynomial interpolation, splineinterpolation, non-linear interpolation (such as a Gaussian process), orother forms of interpolation that will be known to those skilled in theart. The type of interpolation utilized will generate varying results,and some will utilize more time in the generation process.

Another embodiment will now be described with reference to FIGS. 5 and 6b. In this example, the user additionally selects the frontal element320. Therefore, the treatment plan is generated such that the follicularunit implantation sites within the first element, the crown element 335,are generated based on a parameter (for example the density) of thefollicular units in more than one second element, in this case twoelements: the forelock element 325 and the frontal element 320. In thisparticular situation, the parameters of the proposed implantation site610 are based on all fourteen control points Q₀, Q₂, Q₃, P₀, P₁, P₂, P₃,P₄, R₀, R₁, R₂, R₃ and R₄ as depicted by the solid, dashed and dottedarrows (the frontal element 320 has 5 additional control points R₀, R₁,R₂, R₃ and R₄.) The parameter(s) of the proposed implantation site 610may be interpolated from the parameter(s) of the fourteen (14) controlpoints, or a subset thereof.

In the examples indicated above, it can be seen that the referencedcontrol points were all disposed on the periphery of the hair elements,that is, they were contour control points. To further customize thetreatment planning process however, the system may also add specificcontrol points 705 other than contour control points 710 to producespecific effects, such as an implant angle or density, or to achievehair swirl and parting. Individual control points 705 may be placedinside the boundary curve 715 by the user, as shown in FIG. 7 a (the twoblack circles 705 within the boundary curve 715). Any interpolated valueof the implantation site 720 (the unfilled/white circle) will beaffected by both contour control points 710 and the individual controlpoints 705, as illustrated in FIG. 7 a. This additional customizationmay be utilized on a single hair element or in combination with thetechniques described above.

FIGS. 7 b and 7 c illustrate another way in which control points may beutilized to automatically generate a hair swirl (also referred to as ahair whorl). A hair whorl is a patch of hair growing in a circularconfiguration with respect to a point on the scalp, the hairs in a hairwhorl growing in different directions to the surrounding hairs, ineither a clockwise or counter-clockwise direction. Provision of a goodhair transplant relies on the ability to create a natural-looking hairwhorl, and is extremely difficult to create manually. Hair whorls aretherefore an important feature particularly in the crown or vertex areaof the scalp. In this aspect of the application, a method forautomatically creating a hair whorl is provided. In the example of theembodiment of FIGS. 7 b and 7 c, first a hair boundary curve such as ahair patch may be identified (step 755 of FIG. 7 c), as illustrated byhair boundary curve 725 (FIG. 7 b). This identification may be carriedout by the computer, as described above for example, proposing a hairboundary curve based on knowledge of existing hair. In other instances,this identification may be carried out by the user, by selecting from arange of templates, or perhaps by free-hand drawing on a touch screen ortablet device by means of a stylus, for example. In some embodiments,the user identifies the hair boundary curve in a two-dimensional (2-D)plane, and one or more modules of an image processor convert the 2Dinput into 3-D locations on 3-D model of a body surface, by determininga point of intersection between the 2-D location values and the 3-Dmodel of the body surface. Once the hair boundary curve 725 has beenidentified, the computer processor or the user assigns various controlpoints 730 to the hair boundary curve. In some instances, theidentification of the hair boundary curve 725 and the control points 730may be carried out substantially simultaneously. As described earlier,the user may modify these assigned control points 730, or add additionalcontrol points (optional step 760 of FIG. 7 c). The addition of controlpoints may include control points added to the hair boundary curve 725or within an area outlined by the hair boundary curve, as illustrated inFIG. 7 b by additional control points 735. To create a treatment planfor a hair whorl within this hair boundary curve 725, a hair whorlcenter 740 within such boundary is identified. This hair whorl center740 may be identified for example, by utilizing the user interfacefeature such as a hair whorl center icon (not shown), which whenselected may automatically add a hair whorl center 740 to the identifiedhair boundary curve 725. This automatic identification may be influencedby characteristics or genetics of the patient, for example, gender,race, age, where the patient parts his/her hair. Alternatively, thesystem may allow the user to touch the hair whorl center icon withhis/her stylus, and drag the icon inside the hair boundary curve 725,such that a hair whorl center 740 is created at a desired location.Though the above example describes that the hair whorl is disposedwithin the boundary 725, it will be appreciated that hairtransplantation plans may also incorporate plans in which the hair whorlcenter is disposed outside the hair boundary curve or hair patch 725. Nomatter how the hair whorl center 740 is identified, or where it islocated, the user may modify its location. Having identified where thehair whorl center 740 is to be located, the processor may automaticallyassign an initial orientation value to each of the control points 730and additional control points 735 (step 770 in FIG. 7 c). This initialorientation may, for example, be based on the orientation of a virtualline 745 from the hair whorl center 740 to each of the respectivecontrol points 730, 735. This automatic assignment of initialorientation value may additionally take into account at least one ormore features from the 3-D model. For example, the orientation may takeinto consideration where the nose, eyes, eyebrows and/or ears, arelocated on the 3-D model of the patient. Having determined the initialorientation values for the control points 730, 735, the processor mayautomatically generate (step 780) proposed follicular unit implantationsites within the area outlined by the hair boundary curve 725 based onthe location of the control points 730, 735, and their correspondingassigned initial orientation values.

In some embodiments of the application, modification of the initialorientation values of the control points 725, 730 may be desirable. Insome instances, this modification may be automatically carried out bythe processing unit, for example, as shown in FIG. 7 b, increasing theangle of orientation by an angle θ (750) in a clockwise direction withrespect to the virtual line 745 to create a clockwise whorl, or in acounter-clockwise direction from the virtual line 745 to create ananticlockwise whorl. The angle θ (750) may be any angle ranging from 0to 90 degrees, for example 5, 10, 20, 30, 40 or 45 degrees. Thisautomatic modification may be utilized to achieve a particular look onthe patient, based on known outcomes, or perhaps to match one or moreexisting hair follicles, or hair whorls that the patient already has. Inother embodiments, the user may modify the initial orientation value ofthe proposed follicular unit implantation sites. The user may modify theinitial orientation value of each or a subset of control point 730, 735,or modification of the initial orientation value of one control points730, 740 may automatically cause modification of each of the othercontrol points 730, 735.

To achieve a more natural looking hair whorl, the processor may furtherautomatically modify the location and/or orientation values of theautomatically generated proposed follicular unit implantation sitesbased on the distance of the proposed follicular implantation site fromthe hair whorl center 740. To create a more natural-looking appearance,the automatic modification may, for example, comprise providing agreater deviation from the initial orientation value (or a greater angleθ) the further the disposition of the proposed follicular unitimplantation site from the hair whorl center, and a lesser deviationfrom the initial orientation value the closer the disposition of theproposed follicular unit implantation site from the hair whorl center.As illustrated by FIG. 7 b, in addition to generating and displaying tothe user the proposed follicular unit implantation locations, the systemmay additionally be configured to generate and display to the user theproposed hair follicles, follicular units or hair grafts. If such adisplay is generated, the length of the proposed follicles, follicularunits, or grafts may be predetermined by the processor or the user, ormay be input by the user. Such length may be modified to create adesired look. It is also apparent that a treatment plan for more thanone hair whorl may be created on a patient.

Implementation of the selection of hair elements may be provided by alisting of the elements and an associated check box for each element onthe monitor. By checking each of the desired hair elements that the userdesires to be associated with one another, these hair elements can begrouped together, in addition to any control points associatedtherewith. Thus proposed sites may be influenced by all associatedelements and their associated control points.

When multiple hair elements are placed on the same body surface, it ispossible that they may at least partially overlap each other. This isillustrated in FIG. 8, which shows an image of a scalp of patient, onwhich three hair elements have been identified, for example, proposedhair patches 810, 820 and 830. These elements may have been identifiedby the treatment planning system, the user, or a combination of both.Such overlapping elements may be generated, for example, when the useris trying to plan implantation elements which are customized to thepatient, taking into account existing hair elements which already havehair, avoiding elements where hair has already been implanted, oravoiding elements in which implantation is not viable (due topre-existing medical conditions, scars, or other such reasons) forexample. These combinations of elements represent shapes that are notnecessarily provided in the form of templates by the treatment planningsystem. However, an overlapped element may result in the generation, forexample, of proposed implantation sites that have incorrect interpolatedvalues. For ease of explanation the hair elements identified in FIG. 8,they have been schematically replicated in FIGS. 9 a and 9 b. FIGS. 9 aand 9 b illustrate three hair elements 810, 820 and 830, which may, forexample, have been generated such that they fill a bald area of apatient's scalp, the elements around it having already existing hair. Asmore clearly illustrated in FIG. 9 a, these three hair elements overlapeach other to varying degrees. Hair elements 810 and 820 overlap orintersect in the area denoted 915; hair elements 820 and 830 overlap orintersect in the area 925; and hair elements 830 and 810 overlap orintersect in the area 935. Finally, all three hair elements 810, 820 and830 overlap or intersect in area 945.

Let us assume that the user were to indicate that the proposedimplantation sites in the hair element 820 were to be based on theparameters of control points on the contour of, and within, the hairelements 820 and 830. Based on the explanation given above, when aproposed implantation site 960 is generated using interpolationtechniques, the interpolation technique will take into consideration theparameters of control points on the contours of hair elements 820 and830, and of control points within those elements, such as the controlpoint 980, as illustrated. However, control point 980 falls within bothelements 820 and 830, and as such will be considered twice when theinterpolated value of the proposed implantation site 960 is determined.It will be considered once when the values of control points on thecontour and in element 820 are considered, and again when the values ofcontrol points on the contour and in element 830 are considered. If theparameter value generated were, for example, density, it will beapparent that the interpolated value of the proposed implantation sitewill therefore be incorrect (potentially with the density being twice ashigh), because the control point 980 will have been considered twice.For the case of the user requesting that the proposed implantation sitesin the hair element 820 were to be based on the parameters of controlpoints on the contour of, and within, the hair elements 820, 830 and810, it will be apparent that some control points may be considered upto three times, if such control points are located in all threeelements. Processing the interpolated value in this manner may lead to aproposed site plan in which the elements of overlap influence theinterpolated value of site, giving an incorrect, distorted orundesirable value. This is particularly evident if density is theparameter of consideration. The treatment plan in this situation may beproposing a higher density value for proposed implantation site 960 thanwould be expected, potentially leading to a not so natural-looking hairtransplant.

To address this issue, the present application proposes a solution. Forexample, each of the hair elements 810, 820 and 830 may be divided intomultiple segments based on where they intersect with one another, anduse a Boolean operator on those segments to reconstruct the real-estateor floor plan of the combination of the elements 810, 820 and 830without the overlap. This is illustrated in FIG. 9 b, in which the threeelements 810, 820 and 830 are identified, element 810 equal to the sumof segments A, D, E and G; element 820 equal to the sum of segments B,E, F and G; and element 830 equal to the sum of segments C, D, F and G.If it is desired that implantation sites be proposed in all three hairelements 810, 820 and 830, an OR operator may be utilized, thusproviding for interpolated values to be proposed within elementsA+B+C+D+E+F+G. However, if all desired hair has already been implantedinto element 810, a difference (DIFF) operator can be used to excludethat area, thus providing for interpolated values to be proposed withinelements B+C+F only. It will be apparent to the user that this approachfacilitates a proposed implantation site plan in which the elements ofoverlap do not overly influence the interpolated value of implantationsites.

In some situations, a region of overlap may be one hundred percent, asillustrated in FIG. 10 in which element 1030 overlaps completely withanother element which extends beyond the perimeter of element 830. Thistype of situation may result, for example, when the user has alreadyimplanted all desired hairs into the element denoted as 1030, and nowwishes to extend coverage to the outer element denoted by 1040. Adifference operator can once again be used such that interpolated valuesare only proposed for the element denoted 1040, and not the element1030. In this manner more hairs are not implanted into element 1030,only into element 1040, in which hair have not yet been implanted. It istherefore possible to construct any shape for hair transplantation,opening up the opportunities for hair art, similar to tattoos, forexample.

It should be noted that although the points which define the boundariesof the hair patch or the contour points, are all 3-dimensional (3-D)points in a 3-D coordinate system, in order to compute some of theformulae indicated above, and to calculate, for example, thepolygon/element intersection, in one embodiment, the system may firstproject those 3-D contour points on a primary 2-dimensional (2-D) planereconstructed using principle component analysis, a technique well-knownto those in the art. Those projected 2-D polygons having coordinates ina 2-D coordinate system, may then be used for intersection computation.Once the computation has been completed, if required, the 2-Dcoordinates may be projected back into the 3-D coordinate system.

It will be appreciated that although the planning of implantation siteshas been described above, the procedures can equally be applied toharvesting sites. The physician may identify harvesting elements fromwhich hair is to be donated, along with associated control points, andrequest that, for example, the number of hairs in that identifiedelement by reduced, reducing the number of hairs in such a way that thedensity is interpolated to be, for example, 80% of the pre-donateddensity value. The other implementations described throughout thisapplication can equally be applied to harvesting.

When front hairline 1110 and a hair element 1120 are placed close toeach other (see FIG. 11), it is conceivable that the treatment planningsystem may only propose sites within the hair element 1120, and as suchthere will be a visible gap between the hairline 1110 and the boundaryof the hair element 1120. Therefore, according to another aspect of thepresent application, the treatment planning system ensure that there isno space between them, or a least any space that does exist is naturallooking, and conforms to the overall physical appearance of hairdesired. In order to do this, the control points between neighboring oradjacent elements are shared. That is, the control points along thehairline 1110 and the boundary of the hair element 1120 that is adjacentto the hairline 1110 are shared. Hair link points are introduced forthis purpose. To link the boundary of one hair element to another, theuser may drag a control point on the boundary or perimeter of one hairelement and dock it on a control point on the boundary or perimeter ofthe other hair element to “link” them. Disconnection, or the“un-sharing” of two control points may be accomplished by using, forexample, another mouse button (right-click) and dragging to “separate”linked control points. The hair link points may be applied to hairlines,hair patches or the region between different hair elements. In theexample illustrated in FIG. 11, there are three shared control pointsbetween front hairline 1110 and connected hair element (e.g. hair patch)1120, namely control points 1130, 1140 and 1150. These control pointslink the hairline 1110 to the hair patch 1120. In this manner, when oneof the shared control points 1130, 1140, or 1150 is updated (for exampleupdating its position, direction, density, etc.), the linked controlpoint on the other element, hair patch 1120 is also updated. That is, ifthe position of the hairline 1110 is moved, the system automaticallyupdates the position of the boundary of the hair patch 1120 whichadjacent to the hairline 1110. In this manner the user does not have theadditional task of changing separately the hair patch 1120, and canconcentrate on the planning process. This will also ensure that there isno gap and instead there is a smooth transition between close hairs thatbelong to different hair elements.

One of the factors that contribute to the successful outcome of a hairtransplantation procedure is the natural-looking appearance of the fronthairline. According to another aspect of the current application, methodfor generating a natural-looking hairline based on control points, andthe techniques of modulation and interpolation are provided. Thesemethods can be executed substantially automatically, or with assistancefrom the user.

FIG. 12 is a flowchart illustrating an example process 1200 forgeneration of a proposed hairline comprising proposed implantationsites. Process 1200 is carried out by a device, such as an implementingmodule of the processing unit described above, and can be implemented insoftware, firmware, hardware, or a combination thereof. The process 1200will be described in reference to FIGS. 12 and 13.

Initially, in step 1210, control points are identified on the contour ofa base hairline 1330 (as seen in FIG. 13), typically identified on whatis to be the front row or curve of hair on the patient's head, in thedirection of the patient's face, forming a hairline curve. These controlpoints may be identified by the user through the user interface, orgenerated from a free-hand drawing, and/or automatically proposed andgenerated by the treatment planning system. For purposes of explanation,FIG. 13 depicts such initial control points 1310 a, 1310 b, 1310 c, 1310d and 1310 e. These initial control points 1310 a, 1310 b, 1310 c, 1310d and 1210 e are used to generate a base hairline 1330 by any number ofknown techniques in which curves can be generated in the field ofcomputer aided design, for example, by means of interpolation by ahigh-order interpolation algorithm such as Bezier interpolation or cubicinterpolation. In this manner, an arbitrarily-shaped curve, based on aspecified number of control points, in this example the five points 1310a, 1310 b, 1310 c, 1310 d and 1310 e will define the contour of the basehairline 1330. Once the initial base hairline has been proposed, theuser may adjust the location and/or orientation of the control pointsinteractively until the resulting base hairline 1330 is the desiredcurve that the user is seeking. In addition, the user may add additionalcontrol points, and/or adjust the density or spacing between the controlpoints, or add a “randomness” factor, for example, as described in theU.S. Pat. No. 8,104,480.

Once the base hairline 1330 has been generated to the user's initialsatisfaction, in step 1220, the respective location and direction of thefollicular units to be implanted (1320 a, 1320 b, 1320 c, 1320 d etc.)along this base hairline 1330 are automatically determined by theplanning module, along the curve, in order to complete the design of thebase hairline 1330. The proposed implantation sites may be generatedbased on density, spacing and/or randomness parameters provided by theuser through the user interface. Once again, if the proposedimplantation sites are not to the user's satisfaction, the user maymanipulate the control points to facilitate the desired changes in theinterpolated implantation sites of the base hairline 1330. In someimplementations, the interpolated sites of the base hairline may becreated without first generating the hair curve on which theinterpolated sites will reside.

When designing the front hairline or hairline, a physician typicallyuses more than one row of hair to define it. Therefore, designing afinal hairline may comprise designing two or more rows of hair. In step1230 of process 1200 for the generation of a proposed final hairline,the user may provide through the user interface the number of rows to beinterpolated, and the modulation (that is, how the number of hairs ineach subsequent row if reduced) from the base hairline 1330. Asillustrated in FIG. 13, the user in this instance has requested thegeneration of two additional hairlines, and a modulation of about 70%.In this manner hairline 1340 is generated a pre-determined distance(provided by the user via the user interface, or automatically proposedby the planning module) from the base hairline 1330, comprising about30% fewer proposed implantation sites than the hairline 1330. Alsogenerated is hairline 1350 which has about 70% of number of proposedimplantation sites of hairline 1340.

Before the final hairline design is proposed, the planning module mayadditionally execute further steps to aid in the creation of anatural-looking hairline. In step 1250, the spacing between any twoneighboring implantation sites can be calculated, for example, a spacingbetween any two neighbor hairs in each of the rows 1330, 1340 and 1350,as well as the spacing between hairs located in neighbor rows (e.g. rows1330 and 1340, or 1340 and 1350). If the proposed implantation sites arecloser than the identified spacing provided by the user earlier, thelocation of the proposed implantation sites may be automaticallyadjusted to correct this. A randomness factor 1260 may additionally beapplied. Having executed these additional steps, the result is a finalhairline having a pre-determined number of rows of hair, modulation, andrandomness factor.

Another of the factors that contribute to the successful outcome of ahair transplantation procedure is the natural-looking appearance of thedensity of hair throughout the patient's head. According to anotheraspect of the current application, methods (and corresponding processorsystems) for generating a natural-looking interpolated density ofimplantation sites based on control points is provided. These methodscan be executed substantially automatically, or with assistance from theuser.

FIG. 14 is a flowchart illustrating an example process 1400 forgeneration of internal implantation sites based on density interpolationaccording to another aspect of the current application. Process 1400 iscarried out by a device, such as an implementing module of theprocessing unit described above, and can be implemented in software,firmware, hardware, or a combination thereof. The process 1400, ratherthan trying to interpolate internal implantation sites in a3-dimensional coordinate system, projects the 3-dimensional coordinatesof a hair element into a 2-dimensional coordinate system andsubsequently interpolates the internal implantation site in the2-dimensional coordinate system. The interpolated internal implantationsites are then projected back into the 3-dimensional coordinate system,in which they are depicted on the 3-dimensional model of the patient.The example of the proposed process will be described in reference toFIGS. 14, 15 and 16.

One of the initial steps 1410 in the process 1400 is to determinespacing between proposed implantation sites for the hair element (e.g.,hair patch) for which internal implantation sites are to be generated.There are numerous ways in which this can be determined, one of which,according to the present application, is converting density information(which may be provided via the user interface by the user, or calculatedby a processing unit, for example) to spacing information. For example,assuming that the pattern of hair is simulated by triangles, onepossible formula for the conversion of density to spacing informationcould be:

${{Spacing} = \sqrt{\frac{200}{{Density}*\sqrt{3}}}},$

where the spacing of adjacent implantation sites from an implantationsite is based on the Density of the implantation site which they areadjacent to. It will be appreciated that other formulations may beutilized and may be tailored to specific situations. For example, onemay prefer to simulate a pattern of hair by squares rather thantriangles perhaps, thus requiring the formula to be modified to reflectthis information.

Having determined the spacing information, a hair element in the form ofa hair patch is identified. This hair patch comprises an outer boundarycurve, which may be defined by one or more contour points. The hairpatch may comprise conventional shapes such as circular, oval, polygonalor other such shapes, or be customized. Whatever the shape, it isintended that this identified hair patch will be on the patient's head,and as such will not comprise a flat surface, or 2-dimensional (2D)surface, but will comprise a 3-dimensional (3D) element. In thisparticular implementation of this aspect of the current application,step 1420 comprises projecting the 3D contour points which define theouter boundary curve in a 3D coordinate system, onto a 2D image in a 2Dcoordinate system. Methods for executing this step are well-known tothose skilled in the art and will not be described herein. For ease inexplanation, the resulting boundary curve 1510 and contour points 1520a, 1520 b, 1520 c and 1520 d in the 2D coordinate system, as depicted inFIG. 15, will be used. In one implementation, the contour points mayadditionally comprise control points, the purpose of which will bedescribed below.

Step 1430 comprises the generation of candidate implantation sites inthe 2D coordinate system. Once again there are numerous implementationsto accomplish this task that will be apparent to those skilled in theart. One of these is to identify a primary axis 1530, and to move in adirection away from the primary axis 1530, across the proposed internalimplantation area towards the portion of the boundary curve 1510 on theother side of the internal implantation area (in FIG. 15, identified as1540), creating or generating candidate sites 1550 which are depicted inFIG. 15 as white circular features. The primary axis 1530 is anarbitrary axis from which a direction can be proposed. In one embodimentthe primary axis is positioned along or parallel to the longestdimension of the hair patch, thus requiring fewer “lines” of hairs to beproposed in the internal implantation area defined by the boundary curve1510, and potentially requiring less processing time. The candidatesites 1550 represent a proposed site where a follicular unit could beimplanted. The generation of candidate implantation sites in the 2Dcoordinate system may be based on a pattern of sites simulated bysquares, as illustrated in FIG. 15, however, triangular placement, orany other such simulation pattern, is within the scope of thisapplication.

Having generated candidate implantation sites in the 2D coordinatesystem, the planning module utilizes the formulation which determinesthe spacing between proposed implantation sites for the hair patch.However, to do this control points may be required. These control pointsmay be discrete control points, or they may comprise, for example, thecontour points as indicated above. In this particular example, let usassume that the contour points 1520 a, 1520 b, 1520 c and 1520 d serveadditionally as control points, with control points 1520 a and 1520 bhaving associated density values of for example 40 hairs per cm² andcontrol points 1520 c and 1520 d having associated density values of,for example 20 hairs per cm². Along the primary axis 1530, theinterpolated density values for each of the proposed implantation siteswill therefore range from 40 hairs per cm² at 1520 a to 20 hairs per cm²at 1520 d, resulting in interpolated spacing value (step 1440), that isthe spacing value between hairs being smaller at the end closest to 1520a (higher density), and larger at the end closest to 1520 d (lowerdensity). Each of the candidate implantation sites has an interpolateddensity value associated with it, and therefore, has a spacing valuealso associated with it, the spacing value representing the minimumspacing required to achieve the desired density interpolation value.

Referring to FIG. 15, based on the information derived from the controlpoint 1520 a, it may be determined that the candidate implantation site1560 a (shown as a black circular shape) is the closest candidate sitewhich meets the spacing value criteria to achieve the desiredinterpolated value. The other candidate sites between the control pointand the candidate implantation site 1560 a being too close. Havingestablished that candidate implantation sited 1560 a is available forimplantation, a virtual circle 1570 a is drawn around the center 1560 a,the diameter of the circle being determined by the spacing valueformulation for this particular site, having an associated density valuewhich has been determined by interpolation. Step (1450) aids inidentifying those candidate sites that are unavailable for implantation.The candidate sites that fall within the circle 1570 a are consideredtoo close to proposed implantation site 1560 a. The planning module isconfigured to execute operations that identify the next availablecandidate implantation site in the same row as proposed implantationsite 1560 a, that is substantially parallel to the primary axis 1530 andwithin the hair region boundary curve 1510. As can be seen in FIG. 15,this is candidate implantation site 1560 b, which now becomes a proposedimplantation site, where it is intended that a follicular unit beimplanted. As before, this proposed implantation site 1560 b also has aspacing value associated with it, calculated from the formula indicatedabove, based on the associated density value which has been determinedby interpolation. However, as indicated in FIG. 15, the spacing value islarger than that that was associated with proposed implantation site1560 a, and as can be seen, a virtual circle drawn around the proposedimplantation site 15606 has a diameter that is larger than that drawnfor proposed implantation site 1560 a. Once again, the planning moduleis configured to execute operations that identify the next availablecandidate implantation site, in the same row as proposed implantationsites 1560 a and 1560 b, substantially parallel to the primary axis 1530and within the hair region boundary curve 1510. As can be seen in FIG.15, this is candidate implantation site 1560 c, which now becomes aproposed implantation site, where it is intended that a follicular unitbe implanted. This procedure is repeated along the row, and then theprocedure initiated again as the next row of proposed implantation sitesare identified, in a direction moving substantially parallel to, butaway from the primary axis 1530, towards the opposite portion of theboundary curve 1510. In this manner all proposed implantation sites areidentified and generated (Step 1460). It will be apparent from thedescription given above, that there will be more proposed implantationsite identified and generated towards the high density end of the hairpatch, the end where control point 1520 a is located. Fewer proposedimplantation sites will have been identified and generated towards thelow density end of the hair patch, the end where control point 1520 d islocated.

The process is repeated, interpolating spacing values, determiningunavailable site locations and generating proposed implantation sitelocations until the entire hair patch has been populated in this manner.The results of such a process are illustrated in FIG. 16, in whichcontour points 1620, which also serve as control points on the boundarycurve 1610 provide for the proposed internal implantation sites to begenerated.

Once the process has been completed, in step 1470, the 2D contour pointswhich defined the outer boundary curve in a 2D coordinate system, andall the 2D coordinates of the proposed implantation sites (for exampleproposed implantation sites 1560 a, 1560 b, 1560 c and 1560 d areprojected back into the 3D coordinate system. Once again, methods forexecuting this step are well-known to those skilled in the art and willnot be described herein. In this manner, generation of internalimplantation sites based on density interpolation is provided.

Typically, in their area of expertise, physicians tend to know, oralready have an idea, of a look that they are trying to achieve fortheir patient. This knowledge may be based on the current patient, otherpatients he/she has treated, from photographs of others, or otherinformation. Therefore, rather than trying to select from a limitednumber of templates, having to wade through a vast number of templates,or having the processor automatically generate hair elements, the usermay prefer to manually generate the desired hair elements, whether thatbe a front hair line (or hair line) 1710 as illustrated in FIG. 17 a, ora hair boundary curve in a form of a hair patch 1810, as illustrated inFIG. 18 a. An example of a methodology that may be utilized to enablethe physician to provide such an original and customized treatment planis illustrated in FIG. 20 by a flow chart. Manual creation of suchproposed hair elements may be achieved in numerous ways, such as forexample, by free-hand, utilizing, for example, a stylus, pen-like deviceor even a finger in combination with a touch screen, image captureboard, or other such similar device. Alternatively, the user may utilizevarious clicks of an input device such as a mouse in combination withdragging motions to draw the desired hair element. However created, oncethe system has displayed a 3-D model of the body surface to the user(step 2005 of FIG. 20), the user is then able to provide via an inputdevice, two-dimensional (2-D) user input (in the form of a drawing or atrace) of a proposed hair element from a free-hand drawing. Thisproposed hair element may be drawn in or surrounding a bald region ofthe patient's head, or overlay a region with one or more existing hairgrafts therein. Once the system (e.g., a processor) has received thisinformation (step 2010 of FIG. 20), in some embodiments of the presentapplication, it may utilize one or more shape/line recognitionalgorithms to transform each hand drawn line, circle or ellipse, etc.into a smooth geometric shape. The processor may optionally at thisstage remove noise, including noise from wobbles or tremor caused by anunsteady hand motion, to make the lines appear more defined, by means ofone or more smoothing modules. Methods for executing this particularstep are well-known to those skilled in the art and will not bedescribed herein. It should be noted that while a particular stepidentified herein may be generally known to those skilled in the art,the proposed methods as a whole described herein are new and inventive.It should also be noted that the combination of steps of the methods ofthe present application may be performed and/or implemented in anyappropriate order.

Having recorded the user input or trace (e.g. freehand drawing) of theinput device, the processor converts the 2-D trace or user input tolocations on the 3-D model of the patient, in the 3-D coordinate system,and having generated a 3-D trace, it is displayed to the user (step 2015of FIG. 20). It will be appreciated that generation of the 3-D trace maynot appear to be optimal to user, perhaps not spanning as large a regionin the 3-D area as expected, in these type of situations, modificationof the 2-D trace, and subsequent automatic modification of the 3-D tracemay be desirable. No matter how many modifications are required, theuser is able to provide input in two-dimensions and have coordinates ofa line or curve in three-dimensions created or identified by the system.Furthermore, the user may only provide this input on one orthogonalview, and does not necessarily have to provide input in more than oneview to obtain the desired result. As discussed above, one technique bywhich this may be achieved is by projecting the 2-D locations onto the3-D model, and computing the points of intersection thereof, thoughother methods to achieve this will be known to those skilled in the art.The processor may optionally at this stage remove noise.

In order to correctly indicate the proposed hair element on the 3-Dmodel of the patient, in one aspect of the disclosure, the processor maydetermine whether the input received represents an open curve or aclosed loop (step 2015 of FIG. 20). This step may be performed in the2-D plane based on the 2-D user input, or in the 3-D coordinate system.One way in which this may be achieved is by a comparison of the firstand last points on the trace. Assuming that the user provides a singlesubstantially uninterrupted hand motion to create the 2D proposed hairelement, if the first 1715 and last 1720 points of the trace areseparated by more than a predetermined distance, as illustrated in FIG.17 a, it can be assumed that the proposed hair element is intended torepresent a hair line. On the other hand, it will be apparent that ifthe first 1815 and last 1820 points of the trace are separated by lessthan a predetermined distance, or substantially the same, as illustratedin FIG. 18 a, it can be assumed the proposed hair element is intended torepresent a closed loop, or hair patch. The processor generates anddisplays the proposed hair element in the form of a hair line or hairpatch as appropriate. In the event that the hand motion to create the2-D proposed hair element comprises an interrupted hand motion, orcomprise multiple segments, thereby creating a broken line or brokenboundary, the processor may be configured to determine if the distancebetween adjacent points along the curve are more than a predetermineddistance apart from one another, and thereby determine if the brokenline is intended to be a complete line or not. In either case, if thegenerated and displayed proposed hair element on the 3-D model isincorrect, the user may adjust it accordingly.

Having automatically generated the proposed hair element on the 3-Dmodel (step 2020), a plurality of control points are automaticallygenerated by the processor based on the proposed hair element. Thoughdescribed a step that is carried out on the 3-D model, it will beappreciated that this step may be carried out on either the 2-D or the3-D model. There are many ways in which this may be achieved, forexample, by using a 2-D or a 3-D curve approximation algorithm, findinga polygon geometry which provides the best fit to the 2-D or 3-D curveusing as few points as possible. It will be apparent that utilization ofthis technique may allow for further user input, specifying how closelythe polygon should substantially match the user input, and thus allowingthe size of the deviation of the polygon geometry from the 2-D or the3-D curve to be customized. Once the best-fit polygon geometry has beengenerated, the processor may identify key feature points, that is thepoints which when connected form the polygon geometry, and based on theidentified key feature points, generate and display a plurality ofcontrol points, 1725, 1825 (step 2025). The feature points and thecontrol points may or may not coincide. Optionally, additional controlpoints 1730, 1830 can be generated based on the generated control points1725, 1825, using, for example, interpolation techniques/algorithms.Such interpolation techniques include, but are not limited to, forexample, Bezier representation and Cardinal Spline algorithms. Thesecontrol points 1725, 1730, 1825, 1830 each have a corresponding locationvalue associated with them, providing the 3-D location of a plurality ofcontrol points, for example, for a hairline (such as a front hairline)it is desirable to define at least two control points, and for a hairpatch it is desirable to define at least three control points. Theorientation values associated with each of the respective control pointsmay be provided by the user. Orientation values may be provided to eachindividual control point, to all, or to a subset of the control pointscorresponding to a proposed hair element. Alternatively, the processormay assign an orientation value to each of the control points.

In another aspect of the application, the processor may automaticallydetermine an initial orientation associated with each of the respectivecontrol points. This automatic determination may take into account atleast one or more features from the 3-D model. For example, theorientation may take into consideration where the nose, eyes and/orears, are located on the 3-D model of the patient. Alternatively, oradditionally, the initial orientation may be a predetermined angle, forexample, the initial orientations may be assigned such that they aredirected in a direction away from the nose of the patient, and follow adirection down the back of the patient. In another aspect of theapplication, the initial orientation associated with the respectivecontrol points, may be based at least in part on one or more existingfollicular units. As with the previous embodiments, modification of eachor any of the initial orientation values is possible.

With the location and orientation value of each control point, theprocessor is able to automatically generate and display proposedfollicular unit implantation sites or proposed harvesting sites (step2030), based on these plurality of control points. As a result,implantation sites or harvesting sites are generated, which are locatedwithin an area outlined by the hair boundary curve 1810, orsubstantially along the hair line 1710. As illustrated in FIGS. 17 b and18 b, the system and/or processor may additionally be configured togenerate and display to the user the proposed follicular units,follicles or grafts 1750, 1850. If such a display is generated, thelength of the proposed follicular units, follicles or grafts may bepredetermined by the processor or the user, or may be input by the user.Such length may be modified to create a desired look.

The use of such manual or free-hand drawing may additionally be used toguide or influence the generation of proposed harvesting sites,follicular unit implantation sites, guiding or influencing theirlocation, orientation, and/or density for example. This aspect of theapplication is described in relation to FIGS. 19 a and 19 b. FIG. 19 aillustrates a hand-drawn, free-hand 2D drawing of a hair boundary curve1905, defining a hair patch. Should a physician desire a hair whorl becreated, rather than rely on the computer system generating a hair whorlcenter in a desirable location, or relying on himself to adequatelylocate a hair whorl center, he/she may instead choose to provideadditional 2-D user input, identifying for example, several curves 1910which together convey a general pattern, plan, or guide which he/shewould like the proposed follicular implantation sites to substantiallyfollow. In this manner, the user does not necessarily need to identifythe hair whorl center, but with the aid of the additional user input,the curves 1910, is able to define a location where such a hair whorlcenter may be located. The physician can additionally indicate areas ofgreater or lesser deviation in orientation. As illustrated in FIGS. 19 aand 19 b, it can be seen that the physician has drawn the ends of thecurves closest to the “center of the hair whorl” 1915 such that theychange orientation more rapidly per unit area outlined by the hairboundary curve 1905 than those further from the “center of the whorl”1915. It will be appreciated that the additional user input may compriseone or more curves, and the one or more curves may comprisesubstantially straight lines, the term curve in this context isconsidered to cover all such variations.

As described above, the system (including the processor) may utilize oneor more shape/line recognition algorithms that may transform eachadditionally hand drawn line or curve, etc. into a smooth geometric lineor curve. The processor may optionally at this stage remove noise,including noise from wobbles caused by an unsteady hand motion, to makethe curves and lines look more defined, by means of one or moresmoothing modules. Having recorded the user input or trace of the inputdevice, the processor converts the 2-D trace or user input to locationson the 3-D model of the patient, in the 3-D coordinate system. As alsodescribed above, the processor may optionally allow for medicationand/or remove noise at this stage.

Having generated the hair boundary curve 1905 and the additional curves1910 on the 3-D model, control points are automatically generated by theprocessor based on the proposed hair boundary curve 1905, and each ofthe curves 1910. There are several ways in which this can be achieved.As described above, a 3-D curve approximation algorithm can be used tofind a polygon geometry which provides the best fit to the 3-D hairboundary curve 1905 using a few points as possible. With respect to thecurves 1910, in the process of generating the proposed hair curves onthe 3-D model, the processor would have identified each of these curvesas open curves, or lines, rather than closed loops, and as such eachwould have two ends. Referring to FIGS. 19 a and 19 b, it can be seenthat curve 1910 a has a first end 1920 and a second end 1925. Using forexample, a 3-D curve approximation algorithm, the processor isconfigured to find a polygon geometry which provides the best fit to the3-D curve using as few points as possible. In this particular case, the3-D curve can be approximated using a first point at the first end 1920of the 3-D curve 1910 a, a second point at the second end 1925 of the3-D curve, and a third point 1930, approximately mid-way between thefirst and second ends, 1920 and 1925 respectively. These points may beutilized as control points, each having a corresponding location valueassociated therewith.

The orientation values associated with each of the respective controlpoints may be provided by the user. The orientation value may beprovided to each individual control point, or a subset of the controlpoints, such as those associated with each line 1910, and thoseassociated with the hair boundary curve 1905 respectively.Alternatively, a single orientation value may be assigned to all controlpoints. Alternatively, the processor may assign an orientation value toeach of the control points. In one aspect of the application, theprocessor assigns an orientation value for the hair boundary curve 1905in the manner described in relation to FIG. 7 b above. In another aspectof the application, the processor assigns an orientation value for eachcontrol point on each of the hair curves 1910 by determining the angleof a tangent to the curve at each respective control point. The tangentangle illustrated by the arrows in FIG. 19 b. This automaticdetermination may additionally take into account at least one or morefeatures from the 3-D model. For example, the orientation may take intoconsideration where the nose, eyes and/or ears, are located on the 3-Dmodel of the patient, or take into account the orientation of one ormore exiting hair follicle or hair grafts. As with the previousembodiments, modification of each or any of the initial orientationvalues is possible.

With the location and orientation value of each control point, theprocessor is able to automatically generate and display proposedfollicular unit implantation sites, based on the plurality of controlpoints. As a result, for example, implantation sites may be generatedwhich are located within the area outlined by the hair boundary curve1905 and which are influenced by the curves 1910. As illustrated in FIG.19 b, the system and processor may additionally be configured togenerate and display to the user the proposed hair follicles orfollicular units, that may be generally referred to as hair grafts. Ifsuch a display is generated, the length of the proposed hair grafts maybe predetermined by the processor or the user, or may be input by theuser. Such length may be modified to create a desired look.

In another aspect of the invention, the control points and/or the hairboundary curves, and/or hair lines, and/or curves illustrated above inFIG. 19 b, may be subsequently removed from the view of the user,allowing the physician to see schematically, how his/her treatment planwill look on his patient. It will be apparent to the reader that theapplication of free hand drawing or manual generation of the desiredhair lines and/or curves can be combined with any of the othertechniques and planning methodologies described herein in reference tovarious Figures and embodiments of the present application. For example,the physician may utilize free-hand drawing to identify hair elementsthat are to be associated with one another or linked together.

According to yet another aspect of the present application, additionalmethods and systems for improving treatment planning are provided.Having created a plan of proposed implantation sites, the treatmentplanning system may also be able to optimize the treatment process, byenabling the user to plan the number of procedures that may be requiredto obtain the desired result. As described in U.S. Pat. No. 8,388,631, askin tensioner that can conform to a body surface and create tension,can be used to facilitate various procedures on the body surface, forexample, harvesting of follicular units (FUs) from various locations ona body surface. These tensioning devices may come in various sizes andshapes and may be configured to conform to a particular element of thebody surface.

Having generated and displayed on the user interface at least oneimplantation element (or region), but typically several implantationelements, the modeling software may utilize various algorithms togenerate and overlay a representation of one or more region locators or,in some embodiments, skin tensioners to cover the proposed implantationor treatment area. The region locators may be used to hold fiducials ormarkers for directing image-guided systems that may be used inimplementing various methods according to the present disclosure. Theoptimization algorithm may also operate to minimize the total number oftensioners to be used in the treatment procedure, minimize the overlayarea between tensioners, and/or potentially reduce the number ofsessions a patient is required to undergo.

To facilitate this, the user may be required to provide some constraintsfor the region locator or tensioner placement, such as for example, themaximum number of region locators or tensioners they would like to usein the procedure, the type of tensioner, the size of the tensioner, etc.This could be done via the user interface. Optionally, the user canmanually adjust a plan that may be generated by the computer, indicatingthe number and placement of tensioners proposed to be adopted by theuser in executing the proposed treatment plan. For example, the user mayadjust the proposed locations of the tensioners to perhaps avoidaggravating existing scars, perhaps.

FIGS. 21 a and 21 b illustrate the overlay of tensioners that may begenerated to enable the user to execute two different treatment plans.In FIG. 17 a, the proposed implantation element(s) span an area thatsubstantially occupies the bald area of the patient's scalp and requiresthe use of four (4) tensioners 2110 a, 2110 h, 2110 c and 2110 d. On theother hand, in FIG. 17 b, the proposed implantation element(s) spans alesser area, requiring only three (3) tensioners.

It will be apparent to those skilled in the art, that there are manyalgorithms that may be utilized to provide such functionality. In oneexample of this aspect of the current application, the processing unitmay determine the outer contour of the combined hair elements, and basedon that information, and the size of the tensioner, place one or morerepresentations of the tensioner such that the entire area within theouter contour is covered by tensioners. In one embodiment of this aspectof the current application, the processing unit may remove fromconsideration any virtual tensioner which encompasses an area with no ora predetermined limited number of proposed implantation sites.

In another example of this aspect of the current application, the areacovered by the combined hair elements can be virtually divided into gridsections, and a linear algebra cost function may be utilized to optimizethe coverage of the grid points. Utilizing a set of one or moreparameters, including but not limited, for example, to the number oftensioners, the location of tensioner and the size of tensioners, avalue can be assigned which represents an objective value of anyparticular combination of values of the parameters in question. Theobjective value may vary depending upon which of the one or moreparameters is weighted most highly, or has more value in any particularsituation. For example, even though the use of fewer large tensionersmay be possible for a particular treatment plan, the values may beconfigured such that smaller tensioners have a greater value, and assuch the objective value may be found to be higher in that situation.This would, for example, be the case where the treatment plan was to becarried out on a child, or a patient with a small head, and it would beeasier to utilize the smaller sized tensioner or region locator ratherthan the larger ones.

When utilizing this aspect of the current application, the treatmentplan therefore comprises the treatment plan in terms of the proposedsite locations/orientations, and the location/orientation of tensioners.Once the full plan meets the expectation of the physician, user and/orthe patient, the user may then register the treatment plan with anactual patient. In some embodiments, this may be accomplished by usingone or more cameras to identify one or more markers on the patient or adevice on the patient. The marker may be a reflector that is secured tothe patient, an ink mark drawn on the patient, an anatomy of thepatient, the tensioner itself (or any portion thereof). Alternativelythe marker may be a marking on a body surface tensioning device utilizedby the physician in the hair transplantation procedure. The identifiedmarker(s) may be used to determine a position and/or orientation of theimplantation region of on the patient. Fiducial detection algorithms maybe used to locate and refine the position of these markers. It will beapparent that the closer the fiducials/reference markers are to theproposed implantation area, the more accurate the registration of thetreatment plan will be to the patient.

It will be apparent that although the methodology described above asdiscrete steps, one or more steps may be combined or even deleted,without departing from the intended functionality of the embodiments ofthe application. It will also be apparent that the methods describedabove may be performed manually, or they may be partially orsubstantially automated, including performed using robotic systems.Although described in a manner indicating that hair is harvested fromand implanted into the same patient, hair can similarly be harvestedfrom one patient and implanted into another. Alternatively, hair can bereceived from another source and implanted.

The foregoing illustrated and described embodiments of the applicationare susceptible to various modifications and alternative forms, and itshould be understood that the applications as generally disclosedherein, as well as the specific embodiments described herein, are notlimited to the particular forms or methods disclosed, and that manyother embodiments are possible within the spirit and the scope of thepresent applications. Moreover, although individual features of oneembodiment may be discussed herein or shown in the drawings of the oneembodiment and not in other embodiments, it should be apparent thatindividual features of one embodiment may be combined with one or morefeatures of another embodiment or features from a plurality ofembodiments. By way of non-limiting example, it will be appreciated bythose skilled in the art that particular features or characteristicsdescribed in reference to one figure or embodiment may be combined assuitable with features or characteristics described in another figure orembodiment. Applicant regards the subject matter of the application toinclude all novel and nonobvious combinations and sub-combinations ofthe various steps, elements, features, functions, and/or propertiesdisclosed herein. Furthermore, the methodologies described can beapplied to any treatment, and is not limited to hair transplantation.

It will be further appreciated by those skilled in the art that theapplication is not limited to the use of a particular system, and thatautomated (including robotic), semi-automated, and manual systems andapparatus may be used for positioning and actuating the respectiveremoval tools and other devices and components disclosed herein.

While the application has been described in its preferred embodiments,it is to be understood that the words which have been used are words ofdescription and not of limitation. Therefore, changes may be made withinthe appended claims without departing from the true scope of theapplication.

What is claimed is:
 1. A method of planning for transplantation of hairgrafts in a body surface, comprising: receiving two-dimensional (2-D)user input of a proposed hair element from a free-hand drawing;generating, with a use of a processor, and displaying the proposed hairelement on a 3-D model of a body surface on a display device; andautomatically generating a plurality of control points based on theproposed hair element on the 3-D model.
 2. The method of claim 1,wherein generating and displaying the proposed hair element on the 3-Dmodel comprises converting 2-D input values to 3-D locations bydetermining a point of intersection between the 2-D value and the 3-Dmodel.
 3. The method of claim 1, wherein automatically generating anddisplaying the proposed hair element comprises: determining if thereceived 2-D user input corresponds to an open or a closed loop; andgenerating and displaying a hair line if the determined inputcorresponds to an open loop, and generating and displaying a hair patchif the determined input corresponds to an closed loop.
 4. The method ofclaim 1, wherein automatically generating a plurality of control pointscomprises utilizing polygon approximation.
 5. The method of claim 1,further comprising automatically determining an orientation value of atleast one of the plurality of control points.
 6. The method of claim 5,wherein the automatic determination of the orientation is based at leastin part on one or more features of the 3-D model.
 7. The method of claim1, wherein an orientation value of at least one of the plurality controlpoints is determined based on one or more existing follicular units oron further user input.
 8. The method of claim 1, wherein a hair elementcomprises a hairline or a boundary curve.
 9. The method of claim 1,further comprising: automatically generating and displaying proposedfollicular unit harvesting or implantation sites based on the pluralityof control points and the orientations of the plurality of controlpoints.
 10. The method of claim 9, further comprising: receivingadditional 2-D user input in the form of a free-hand drawing; andwherein the automatically generated proposed follicular units sites areimplantation sites and wherein orientation of the automaticallygenerated proposed follicular units at the implantation sites is basedon the additional 2-D user input.
 11. The method of claim 10, whereinthe proposed element is a hair patch, and the additional 2-D user inputcomprises one or more curves drawn within an area outlined by a boundaryof the hair patch.
 12. The method of claim 11, further comprisingautomatically generating at least two control points based on each ofthe one or more curves.
 13. The method of claim 12, wherein one of theat least two control points comprises a control point at one end of eachof the one or more curves, and another of the at least two curve controlpoints comprises a control point the other end of each of the one ormore curves.
 14. The method of claim 12, further comprisingautomatically determining orientation of each of the at least twocontrol points, wherein the automatic determination comprisesdetermination of an angle of a tangent to the curve on the 3-D model atthe location of each respective control point.
 15. The method of claim12, further comprising automatically generating and displaying proposedfollicular unit implantation locations based on the plurality of controlpoints and the orientations of the plurality of control points, and onthe at least two control points and the orientations of the at least twocontrol points.
 16. The method of claim 1, further comprising removingnoise from the free-hand drawing.
 17. The method of claim 1, wherein thehair element comprises a hair patch, and further comprising identifyinga location of a center of a hair whorl with respect to a boundary of thehair patch.
 18. The method of claim 17, wherein the hair whorl islocated within the boundary of the hair patch.
 19. The method of claim18, wherein the automatically generating a plurality of control pointsfurther comprises automatically generating an orientation value for eachcontrol point, the orientation value being based on a virtual line drawnbetween the center of the hair whorl and the each of the plurality ofcontrol points.
 20. The method of claim 19, further comprising alteringthe orientation value assigned to the plurality of control points basedon a distance of the control point from the center of the whorl.
 21. Themethod of claim 1, further comprising displaying a three dimensional(3-D) model of a body surface on a display device.
 22. A method ofautomatically creating a plan for a hair whorl on a body surface,comprising: identifying a hair boundary curve, the hair boundary curvehaving a plurality of control points; identifying a hair whorl center;automatically assigning an initial orientation value to each of theplurality of control points, the initial orientation value based on theorientation of a virtual line from the whorl center to each of therespective plurality of control points; and automatically generatingproposed follicular unit implantation sites within an area outlined bythe hair boundary curve based on the plurality of control points andtheir corresponding assigned initial orientation values.
 23. The methodof claim 22, wherein the hair whorl center is inside the hair boundarycurve.
 24. The method of claim 22, further comprising modifying anorientation of an automatically generated proposed follicular unitimplantation site based on its distance from the hair whorl center. 25.The method of claim 24, wherein modification of the orientation of theautomatically generated proposed follicular unit implantation site issuch that it has a greater deviation from the initial orientation valuethe further it is from the hair whorl center, and a lesser deviationfrom the initial orientation value the closer it is to the hair whorlcenter.
 26. The method of claim 22, wherein identification of the hairboundary curve is carried out by a free-hand drawing implement.
 27. Themethod of claim 26, wherein the hair boundary curve comprises a 2-Dcurve, and the method further comprises converting the 2-D curve to 3-Dlocations by determining a point of intersection between the 2-D valuesand a 3-D model of a body surface.
 28. The method of claim 22, whereinautomatically assigning an initial orientation value of each controlpoint further comprises taking into account at least one or morefeatures of a 3-D model.
 29. The method of claim 22, further comprisingmodifying the initial orientation values assigned to the plurality ofcontrol points.
 30. A method of planning for transplantation offollicular units in a body surface, comprising: receiving one or moretwo-dimensional (2-D) curves; automatically assigning at least twocontrol points to each of the one or more curves; automaticallydetermining an initial orientation value for each of the control pointsbased on an angle of a tangent to the one or more curves at eachrespective control point; and automatically generating proposedfollicular unit implantation or harvesting sites based on the at leasttwo control points of the one or more curves and their correspondingautomatically assigned initial orientation values.